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• ABSTRACT 

In light of current research and evaluation criteria, 
the Gallup Evaluation Model (GEM I and GEM II) was developed and 
tested at Pennsylvania State University to assess effectiveness and 
efficiency of instructional developments* Alternative ways of 
teaching German I were evaluated according to five variables: amount 
. of pupil time, course costs, number of students, student achievement 
and attitude. A cost-ef f ect* veness analysis compared student 
achievement and attitude scores to costs of each alternative 
instructional method. The Pennsylvania study concluded that the model 
is practical, readily applicable, and workable for deciding which 
program to eliminate or revise. It is also useful in predicting 
changes in effectiveness during course recycling and when reworking 
certain specific variables. (SK) 
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INTRODUCTION 



In the past several years education has seen many rapid changes. 
Innovations in instruction have flourished. Many of these innovations 
have been widely adopted. One such innovation is the application of a 
systematic approach to instruction incorporating the use of mediated 
instructional materials. The primary purpose of this innovation has 
been to enable the most efficient and effective instructional methods to 
be utilized in the classroom. Tne overall goal of this instructional 
practice has been to provide instruction to larger numbers of students, 
in less time and at a lower cost, while at the same time providing a 
higher quality of instruction. 

Within the last several years, instructional development projects 
and many specific, step-by-step models for the process of instructional 
development have been devised. Table 1 lists some of these early 
instructional development models, their authors, and dates of development. 
Table 1. Instructional Development Models. (Twelker, Urbach, Buck, 



1972) 



Title 



Author 



Date 



1 . Teaching Research System 



Hamreus 



1968 



2. Michigan State University Instructional 
Systems Development Model 



Barson 



1967 



3. System Approach for Education (SAFE) 



Corrigan 



1966 



4. Project MINERVA Instructional System 
Design 



Tracey 



1967 



5. Banathy Instructional Development System Banathy 



1968 
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Recently in 1972 the Pennsylvania State University initiated a 
"Program for the Improvement of Instruction" which supported innovative 
redevelopment of basic courses that served students at several campus 
locations. The Program funded selected courses for instructional 
development over the time span of one year. The intent was to improve 
both instructional effectiveness and cost efficiency. However, the 
question had been raised, as a result of the investment of time, money 
and materials in the instructional development of a course, was 
instruction more effective and/or efficient? 

The University Division of Instructional Services at the Pennsylvam 
State University, which became involved in the production aspect of the 
Program for the Improvement of Instruction, had several criteria to 
evaluate outcomes of the redeveloped courses. These criteria as listed 
by Greenhill (1972) were: 

CRITERIA FOR EVALUATION OF OUTCOMES 
Cost/benefits 

1. Same cost but improved quality - (e.g., 
fewer drop outs, improved, learner com- 
petencies, improved attitudes toward 
course). 

2. Handle more students at same cost. 

3. Cover more material (which is learned) 
in same time. 

4. Cover same amount of material in less 
time. 

5. Make more efficient use of space. 

6. Have students with lower or more varied 
input competencies reach desired level 
of performance. 

While the criteria for the evaluation of a course selected for 
instructional development were specific and detailed, no method was 



provided for combining these factors into' an organized procedure for 
assessment and -decision making. It seemed as though some procedure 
would be useful in making an objective, quantitative evaluation of 
the outcomes of a redeveloped course. 

Statement of the Problem 
As a result of the systematic approach, i.e., instructional 
development being utilized with increased regularity to design 
instructional strategies, some procedure is required to determine the 
overall combined impact of all of the instructional variables such as 
those Greenhill lists as criteria to evaluate the outcomes of redeveloped 
courses. This procedure would provide information necessary in the 
evaluation of the instructional process and assist in deciding whether 
to revise or not revise a course of instruction. Further, this 
procedure would help answer the questions: Is the product of instructional 
development effective? Does it contribute to improved learning? If 
so, how much? If not, why? 

Purpose of the Study 
The purpose of this study was to develop and implement a model 
which could be used in the evaluation process for comparing instructional 
methods. The model attempted to determine the relative effectiveness 
of two different methods or alternatives of teaching the same course 
content. The first method of instruction was defined as the established 
or traditional instructional process. The second method was defined as 
any set of educational strategies which were developed by following 
an organized sequence of course development, i.e., an instructional 
development process. For this study, efficiency and effectiveness 



of a course were defined as a function of the following variables: 
(1) student achievement, (2) student attitude, (3) number of students 
in a course, (4) cost of offering a course, and, (5) amount of time 
students spend in course instruction. 

The development of a model such as the one proposed in this study 
had been recognized as a fruitful area for consideration by instructional 
developers concerned with the need to optimize instructional methods and 
evaluate the results of instructional development. Davies (1971) stated, 
"In view of the increasing investment now being made in educational and 
training systems, some form of responsible audit or evaluation is becoming 
more and more necessary." It seemed that some method of determining 

« 

efficiency and effectiveness was becoming more and more necessary to make 
decisions about different instructional methods. 

REVIEW OF THE LITERATURE 

A survey of the literature revealed that an adequate evaluation model 
has been sought by educators for some time. A collection of models, 
flowcharts, and diagrams have emerged to answer this demand, i.e., Alkin 
(1969), Stufflebeam (1957), Klein (1972), Stake (1967) and Greenhill (1972) 
However, few of these models have taken-into-account all the factors 
necessary in assessing practical improvement in a course as a function 
of change in method of instruction. Research which involved the formative 
evaluation of an instructional development process was also limited and 
provided further impetus for this study. 

No studies were located which compared a method of instruction 
resulting from an instructional development process with another form of 
teaching strategy development. However, many articles, reports, and books 
have been written concerning the instructional development process, includi 



books by: Lee (1971). Kemp (1971), Gustafson (1973), Banathy (1968), 
Crawford (1969), and Diamond (1971). 

MODEL DESIGN AND IMPLEMENTATION 
INTRODUCTION 

A theoretical model developed by Stephen Klein (1972) was located 
which had potential for use in the formative evaluation of two methods 
of teaching the same course content. However, several modifications 
were necessary to make this theoretical model practical and workable as 
an instrument in the formative evaluation process. 

Based on Klein's work, two new models for evaluating and comparing 
instructional programs were developed. Hereafter, these two models will 
be referred to as the Gallup Evaluation Model I (GEM Model I) and the 
Gallup Evaluation Model II (GEM Model II). One of these models assesses 
course effectiveness on an achievement basis (GEM Model I) while the other 
models take into consideration the interaction between most of the 
variables listed by Greenhill (1972) as criteria to evaluate the 
outcomes of redeveloped courses. 

The GEM Model I: Achievement Basis 
In determining course effectiveness on an achievement basis. Total 
Course Effectiveness may be represented by E^ . Effectiveness of one 
instructional method (Alternative I) may be represented by e^ . The 
other instructional method's effectiveness (Alternative II) as 
Total Course Effectiveness (E-,) = Effectiveness of Alternative I (e-,) + 
Effectiveness of Alternative II (e2). The effectiveness of an alternative 
(e^ or may be viewed as the achievement gained in al alternative 
multiplied by the number of students enrolled in an alternative. 
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divided by the product of the time stucents devoted to instruction 

multiiilied by the' costs of offering an alternative or: 

= (A) X (S) 
^ ID ^ CcT 

The development of the model may be understood more clearly by 
examining information in Figure 1. Assuming a course is made up to 
two sections, and each section is taught by a different alternative 
method of instruction, a measure of effectiveness for each alternative 
may be determined, as desc'- ioed above. Then, a percentage of Total 
Course Effectiveness (E-,) can be calculated for each section: 

E^ = e^ + ^2 

Relative effectiveness pf 

Alternative I = e x 100 



Relative effectiveness of 

Alternative II = eg x 100 



Once a percentage of relative effectiveness is calculated for each 
alternative, a decision may be reached concerning which alternative 
method of instruction should be implemented, which method should be 
eliminated and/or which method should undergo revision. 

For the purposes of this model the following variables are used in 
determining relative effectiveness on an achievement basis: 

Amount of Pupil Time . The amount of time is determined by the 
amount of time students devote to a course, both in and out of class, 
i.e., direct instruction time, textbook or supplemental reading time, 
laboratory time, library time, writing time, time spent in studying for 
quizzes and examinations, as well as the time involved in taking quizzes 
and examinations. For the purposes of GEM Model I, time students devote 
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to a course is recorded on a time sheet in hours and minutes on an 
individual basis. 

Costs of Offering a Course. The costs of offering a course includes 
all those expenses incurred in conducting a course. These costs include, 
the proportion of faculty and staff salaries devoted to each course, the 
costs of space in which a course is taught (costs of classroom space 
including utilities), and the expendable costs such as stencils, reproduction 
materials and paper. Faculty and staff salary figures are obtained from 
the specific individuals involved. The cost of classroom space is obtained 
through University sources. Costs of expendable materials are kept by the 
instructors on expense forms. Costs of resources utilized by a large 
majority of a University community are not included. These include 
such facilities as University libraries, learning centers, and language 
laboratories. 

Number of Students . The number of students is determined by the 
enrollment within the sections of a course. For purposes of GEM Model 
I, all students in the treatment sections completing both the pre- and 
post- tests are considered as the number of students enrolled. 

Student Achievement . Student achievement is determined by the 
administration of a standardized achievement test selected from a source 
such as the Euros' Mental Measurements Yearbook on the basis of course 
subject matter. Students in all treatment sections receive this 
standardized test both before and after instruction. The score used for 
calculating effectiveness, utilizing GEM Model I, is the gain score 
difference between pre- and post- tests. 

GEM Model II: Attitude Basis 
In determining course effectiveness on an attitude basis. Total Course 
Effectiveness is represented by E«. Effectiveness of Alternatives I and 



II are represented by and e^ respectively. Total Course Effectiveness 
(E2) = Effectivenass of Alternative I (e^) + Effectiveness of Alternative 
II (62). The effectiveness of an alternative (e-, or e2) may be viewed as 
the total positive attitude in an alternative multiplied by the number of 
sfjdents enrolled in an alternative, divided by the product of the time 
students devote to instruction multiplied by the costs of offering an 
alternative, or: 

e, = (At) X (S) 

The development of the model may be understood more clearly by 
examining information in Figure 2. 

Assuming a course is made up of two sections, and each section is 
taught by a different alternative method of instruction, a measure of 
effectiveness of each alternative may be determined as described above. 
Then, a p^.xentage of Total Course Effectiveness (E^) can be calculated 
for each section: 

E^ = -f e2 

Relative effectiveness of = e^ x 100 
Alternative I 

Relative effectiveness of e^ x 100 
Alternative II 

Once a percentage of relative effectiveness is calculated for each 
alternative a decision may be reached concerning which alternative method 
of instruction should be implemented, which method should be eliminated 
and/or which method should undergo revision. 

For the purposes of this model the variables used in determining 
effectiveness on an attitude basis are the same as those employed in the 
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determination on an achievement basis with the exception of the achievement 
variable. The achievement variable is eliminated and replaced by an 
attitude variable. The value of this variable is the total positive score 
obtained by students on a standardized course attitude survey administered 
upon course completion. 

MODEL IMPLEMENTATION 
The German 1 course at The Pennsylvania State University, undergoing 
instructional development during the Summer and Fall terms of 1973, was 
selected for the trial implementation and testing of the developed models 
in the formative evaluation process. The study sample consisted of college 
students enrolled in Sections 1,2, 4, and 5 of German I during the Winter 
Term, 1973-1974. Alternative I, the on-going, established instructional 
method was utilized in Sections 4 and 5. Alternative II, a prototype 
form of instruction resulting from an instructional development process, 
was employed in Sections 1 and 2. Two instructors each taught two sections 
of German 1, one section by Alternative I, the other section by Alternative II. 

FINDINGS 

As a result of this trial implementation of the developed GEM Models 
in the formative evaluation of two methods of teaching the same course 
content, the following findings are presented. 

It was possible to develop evaluation models v;hich were practical, 
workable procedures for use in the evaluation of two methods of teaching 
the same course contents. The two models which were developed both on an 
achievement and attitude basis brought together factors necessary in 
assessing course effectiveness and efficiency. 

A second finding was the ease with which the developed models could 
be implemented in the evaluation process. The data for the factors of 



number of students, student achievement, and student attitude would normally 
be gathered in any reliable evaluation of instruction. Costs of offering 
a course and amount of pupil time are two factors requiring data gathering 
which may not normally be done in a formative evaluation of course insturction 

When implementing these models as part of the evsluation process, 
differences in effectiveness and efficiency resulting from two ways of 
teaching the same course content were detected. In detecting these 
differences the GEM Models examine the interaction between variables 
associated with course instruction. These variables, which include Student 
Achievement and Attitude, Time Spent in Course Instruction, the Costs of 
Offering Instruction, and the Number of Students enrolled in instruction, 
make up the criteria Greenhill (1972) listed as necessary to evaluate 
redeveloped courses. The differences in effectiveness and efficiency 
which are detected may determine whether a method of instruction is 
implemented, eliminated, or where revisiojj can be made within a method. 

DISCUSSION OF FINDINGS 

The GEM Models which were developed for use in evaluation process had 
a main advantage and a corresponding disadvantage. The advantage of the 
GEM Models was that they brought together variables which were indicators 
of course effectiveness and/or efficiency, particularly criteria for 
evaluation as listed by Greenhill (1972), in a practical, workable procedure 
for use in the formative evaluation process. The disadvantage occurs 
because the GEM Models did not examine specific variables alone. If the 
priority of goal of two methods of teaching the same course content was 
to improve student achievement, then the GEM Models were at a disadvantage 
becuase they could not detect a change in a single variable. 

To illustrate this disadvantage, the formative evaluation of German 1 
on an overall achievement basis is used as an example. Students in 



Alternative II gained an average 1.35 points more on the achievement 
measure than students in Alternative I. However, overall Alternative I 
reached a higher percentage of relative effectiveness (58.80% vs. 41.19X) 
primarily because students in Alternative II devoted an average of 16 hours 
more to course instruction. In this example, the larger amount of time 
in Alternative II offset a gain in achievement by students in Alternative 
II, making Alternative I relatively more effective (+17.61% higher). 
Therefore, it should be kept in mind that while the GEM Models do bring 
together variables which are indicators of course effectiveness, these 
same variables should be examined by other evaluation instruments on an 
individual basis. 

The five variables incorporated in the GEM Models each had specific 
concerns which may need to be taken into consideration when implementing 
them in the evaluation process. The five variables and a discussion of 
their specific concerns follow. 

Amount of Pupil Time . The amount of time students devoted to a 
course was viewed as a negative variable (Klein, 1972). One criteria 
listed by Greenhill (1972) was to cover the same amount of material in 
less time. Less time devoted to one course allowed students to spend 
time on other courses or activites. Certain concerns should be kept in 
mond regarding the amount of pupil time variable incorporated in the 
developed model. One concern was whether the amount of time recorded by 
students was accurate. In this study it was assumed that students recorded 
the actual amount of time they devoted to course instruction. 

Costs of Offering a Course . The costs of offering a course was the 
other negative variable. The main concern associated with this variable 
was in obtaining all costs incurred by an alternative method of instruction. 
If all costs were correctly determined, the value of specific expenses 



may be used in the redevelopment of a method of instruction to maka that 
method more efficient. 

Number of Students . The number of students was viewed as a positive 
variable. The major concern regarding the number of students is that a 
method if instruciton enrolls the largest number of students to which it 
can effectively provide course instruciton. This results in an optimum use 
of classroom materials and space. A low number of students enrolled in a 
course results in poor utilization of the costs of offering a course. 

Student Achievement and Student Attitude . Student achievement and 
student attitude were the other two positive variables. The main concern 
with these variables was that slight gains in them were offset by larger 
gains in the negative variables of amount of pupil time and costs of 
offering a course. 

It must be kept in mind that the models were not developed to examine 
specific variables by themselves. Decisions regarding student achievement 
or attitude made utilizing the models may be misleading and/or ^iased. 
The developed GEM Models examine these variables in combination with other 
variables and put them is a perspective to determine overall course 
effectiveness and/or efficiency. 

CONCLUSIONS 

Based on the findings of this study the following conclusions were 
drawn regarding the GEM Models I and II. 

1. The Models are workable procedures for the evaluation of two 
methods of teaching the same course content. Based on Scriven (1967) the 
developed models provide a means or evaluating courses in a state of 
development. Bloom, Hastings, and Madaus (1971) state: "learning is a 
process which can be observed and evaluated as it takes place." The models 
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developed in this study observe and evaluate a learning process as it is 
occurring. 

2. The Models are only one instrument in the evaluation process. 
Based on the data analyzed by the models in this study it would appear at 
first glance that Alternative II, or the alternative that underv/ent 
instructional development, has proved to be less effective than Alternative 
I. However, in looking at mean achievement and attitude scores (Table 12) 
which are also indicators of course success, students in Alternative II 
reached a higher level of achievement (+1.76 points) and at the same time 
were slightly more favorable toward Alternative II as a method of instruction 
(+3.77 points). Another source of evaluation may come from people directly 
involved with the alternatives of instruction— course instructors. Based 

on the observations made by the instructors of German 1, students in 
Alternative II had better class attendance (resulting in a larger time 
variable— +16 hours), participated in class discussion more freely, and 
seemed to enjoy class more than students in Alternative I (as indicated 
by a larger attitude variable score~+3.77 points). 

3. The Models examine and take into consideration factors of 
educational effectiveness and efficiency. Utilizing the variables amount 
of pupil time, costs of offering a course, number of students, student 
achievement, and student attitude in the evaluation process, a basis may 
be established for making decisions involving the most efficient and 
effective method of course instruction. 

4. The Models may be used for predicting effectiveness of outcomes 
during a recycling of the course development process. If a particular 
alternative of instruction has shown a weakness in one variable during an 
initial evaluation, course developers can determine the specific change 
needed to strengthen the identified variable. As an example, in this 



study the cost of Alternative II was $84.13 more than Alternative I. If this 
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larger cost is considered a weakness and warrants improvement, reducing 
the cost variable of Alternative II by $84.13 can be recalculated to predict 
a new relative effectiveness for each alternative. This prediction ability 
of the models may help instructional developers identify specific variables 
which, with reworking, will improve both the effectiveness and efficiency 
or a method of course instruction. 

5. The Models may only be used to interpret those variables which 
were in existence at that point in time when the formative evaluation 
occurred. Decisions regarding course recycling or redevelopment should 
be made based upon model findings and not upon what might have been. In 
course recycling the models may be used to predict changes in effectiveness 
and efficiency resulting from reworking specific variables. 

6. The Models should only be used when both methods for teaching 

the same course content have as their overall goal the same basic objectives. 
When two methods of teaching the same course content have overall goals 
which are the same, these two models may be used for evaluation. As an 
example, the overall goals for both alternatives of instruction in this study 
were to teach students the basic readiru^, writing, listening, and speaking 
skills of elementary German. While the instructional methods were different, 
the overall objectives were the same. 
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